Title: Learning to see and hear without human supervision
Date: February 16th, 2021 at 10:00 am
Place: Virtual meeting
Presenter: Pedro Morgado
Abstract:
Imagine the sound of crashing waves. This sound may evoke the image of a beach. A single sound serves as a bridge to connect multiple instances of a visual scene. It can group scenes that ‘go together’ and set apart the ones that do not. Audio can thus serve as a target to learn powerful representations for visual inputs without relying on costly human annotations. As computer vision systems become more capable, human annotations become the bottleneck for further developments. My goal is to develop effective training procedures that curb the need for direct human supervision.
In this talk, I will discuss several tasks that benefit from audio-visual learning, including representation learning for action and audio recognition, visually-driven sound source localization, and spatial sound generation. I will introduce an effective contrastive learning framework that learns audio-visual models by answering multiple-choice audio-visual association questions. I will also discuss important challenges we face when learning from audio supervision related to the presence of noisy audio-visual associations, and how to overcome these challenges using robust learning algorithms.
Bio:
Pedro Morgado is a Ph.D. candidate in the Electrical and Computer Engineering department at the University of California, San Diego advised by Prof. Nuno Vasconcelos. He has also spent time at Adobe Research working with Oliver Wang and Facebook AI Research working with Ishan Misra. His main research interests lie in computer vision and deep learning, focusing on multi-modal self-supervised learning. His work aims to develop algorithms that make the power of computer vision accessible by lowering the two major costs of deep learning – the dependence on human annotations and the high compute requirements of training and deployment. Pedro is the recipient of a 4-year graduate scholarship from the Portuguese Science and Technology Foundation. Before arriving to San Diego, he received a Bachelor’s and Master’s degree at Instituto Superior Técnico in Lisbon, Portugal.
Host: Adriana Kovashka
